aq method concept drift detect insid theft trade secret pair learner concept drift elicit system detect insid violat dto dknow learn detect classifi malici execut wild describ machin learn data mine detect classifi malici execut wild gather benign malici execut encod train gram byte code featur process result million distinct gram select relev gram predict evalu varieti induct method includ naiv bay decis tree support vector machin boost ultim boost decis tree outperform method area roc curv result suggest methodolog scale larger collect execut evalu method classifi execut base function payload open backdoor mass mail area roc curv detect payload function neighborhood smaller detect task attribut drop perform fewer train exampl challeng obtain properli label exampl fail methodolog inher difficulti classif task final appli detector malici execut discov gather origin collect boost decis tree achiev true posit rate desir fals posit rate result import suggest methodolog basi oper system detect previous undiscov malici execut addit expert ensembl cope concept drift onlin learn target concept chang time previou work expert predict algorithm bound worst case perform subsequ train data rel perform expert expert difficult implement gener approach bound perform rel actual perform onlin learner singl subsequ present addit expert ensembl algorithm addexp gener method onlin learner drift concept adapt techniqu analyz expert predict algorithm prove mistak loss bound discret continu version addexp final present prune method empir result data set concept drift learn detect malici execut wild describ machin learn data mine detect classifi malici execut wild gather benign malici execut encod train gram byte code featur process result million distinct gram select relev gram predict evalu varieti induct method includ naiv bay decis tree support vector machin boost ultim boost decis tree outperform method area roc curv result suggest methodolog scale larger collect execut evalu method classifi execut base function payload open backdoor mass mail area roc curv detect payload function neighborhood smaller detect task attribut drop perform fewer train exampl challeng obtain properli label exampl fail methodolog inher difficulti classif task final appli detector malici execut discov gather origin collect boost decis tree achiev true posit rate desir fals posit rate result import suggest methodolog basi oper system detect previous undiscov malici execut increment learn partial instanc memori learn concept chang time import varieti applic intellig system acquir behavior profil comput intrus detect calendar schedul intellig user interfac exampl interest class method learn concept consist algorithm maintain portion previous encount exampl concept chang time method store select exampl mechan exist identifi remov irrelev exampl concept paper describ increment rule learner partial instanc memori call aq pm wah widmer kubat heurist adjust dynam window retain forget exampl evalu learner stagger concept made direct comparison aq pm aq pm similar learner partial instanc memori result suggest forget heurist restrict flora learner origin design result studi suggest learner partial instanc memori converg quickli chang target concept algorithm learn sole exampl dynam weight major ensembl method track concept drift algorithm track concept drift import applic present gener method base weight major algorithm line learner concept drift dynam weight major dwm maintain ensembl base learner predict weight major vote expert dynam creat delet expert respons perform empir evalu experiment system base method increment naiv bay increment tree induc iti expert gener model finit dsampl effect train test compet classifi convent wisdom field statist pattern recognit spr size finit test sampl domin varianc assess perform classic neural classifi present work show result narrow applic compet algorithm compar finit train sampl commonli domin uncertainti gener problem spr analyz formal structur recent develop multivari random effect receiv oper characterist roc analysi mont carlo trial gener model explor detail statist structur repres problem field computeraid diagnosi medicin scale law varianc accuraci measur number train sampl number test sampl investig found compar discuss classic text fukunaga import interact term neglect previou author final import contribut finit trainer uncertainti argu form bootstrap analysi sampl uncertainti lead contemporari candid extens bootstrap error analysi oppos commonli cross valid improv rooftop detect aerial imag machin learn paper examin machin learn improv rooftop detect process step vision system recogn build overhead imageri review problem analyz aerial imag describ exist system detect build imag briefli detail algorithm select improv rooftop detect data set highli skew cost mistak differ class roc analysi evalu method vari error cost report experi design illumin facet appli machin learn imag analysi task investig learn imag determin perform method focus imag learn deriv train test data imag final experi address imag learn train test set imag result suggest gener occur train test data deriv imag differ locat aspect demonstr condit naiv bay exceed accuraci method handcraft classifi solut build detect system machin learn roc analysi statist test signific roc analysi greater frequenc evalu methodolog machin learn pattern recognit research anova determin result analysi statist signific medic decis make commun prevail method labmrmc method anova appli jackknif method account case sampl varianc determin test make decis statist signific conduct mont carlo simul problem deriv gaussian distribut machin learn algorithm roc analysi anova labmrmc result suggest decis test make simpl problem larger issu anova account case sampl varianc gener experiment result popul data drawn increment learn partial instanc memori learn concept chang time import varieti applic intellig system acquir behavior profil comput intrus detect calendar schedul intellig user interfac exampl interest class method learn concept consist algorithm maintain portion previous encount exampl concept chang time method store select exampl mechan exist identifi remov irrelev exampl concept paper describ increment rule learner partial instanc memori call aq pm wah widmer kubat heurist adjust dynam window retain forget exampl evalu learner stagger concept made direct comparison aq pm aq pm similar learner partial instanc memori result suggest forget heurist restrict flora learner origin design result studi suggest learner partial instanc memori converg quickli chang target concept algorithm learn sole exampl initi studi adapt hierarch vision system describ empir studi adapt hierarch vision system simpl vision task requir low level high level process examin scheme feedback line learn affect true posit rate number instanc learn user feedback rst scheme learn instanc user provid feedback instanc assum feedback meant correct classic end hybrid scheme method dierent level yield result show exampl learn signicantli improv true posit rate classier lower level higher level hybrid method increas user feedback introduct hierarch vision system mohan amp nevatia work repeatedli group visual construct higher level construct select select exampl partial memori learn paper describ method select train exampl partial memori learn system method select extrem exampl lie boundari concept descript exampl train exampl induc concept descript forget mechan activ remov exampl partial memori irrelev outdat learn task implement method conduct lesion studi direct comparison examin effect partial memori learn predict accuraci number train exampl maintain learn experi involv stagger concept synthet problem real world problem blast cap detect problem comput intrus detect problem experiment result suggest partial memori learner notabl reduc memori requir slight expens predict accuraci track concept drift ot modifi rete reason tempor 